Supplementary Material for Learning Robust Visual-Semantic Embeddings

Fig. 1 provides an easy-to-understand design of ReViSE. In all of our experiments, GoogLeNet is pre-trained on ImageNet [2] images. Without fine-tuning, we directly extract the top layer activations (1024-dim) as our input image features followed by a common log(1+v) pre-processing step. For the textual attributes, we pre-process them through a standard l2 normalization. In ReViSE, we set α = 1.0 in eq. (11), so that we place equal importance on supervised and unsupervised objectives. For the visual auto-encoder, we fix the parameter of the contraction strength γ = 0.1 in eq. (2). In the following, we omit the bias term in each layer for simplicity. The encoding of visual features is parameterized by a twohidden layer fully-connected neural network with architecture dv1−dv2−dc, where dv1 = 1024 is the input dimension of the visual features, dv2 = 500 is the intermediate layer, and dc denotes the dimension of the visual codes ṽh. To encode textual attributes, we consider a single-hidden layer neural network dt1−dc, where dt1 is the input dimension of the textual attributes. We choose dc = 100 when dt1 > 100 and dc = 75 when dt1 < 100. Furthermore, we do not tie the weights to be learned between the decoding and encoding parts. Parameters for associating distributions of visual and textual codes (MMD Loss) in eqs. (5) (12), and (6) are set as β = {0.1, 1.0} (chosen by cross-validation) and κ = 32.0. For the remaining part of our model, we set the architecture of visual and textual code mapping as a single-hidden layer fully-connected neural network with dimension dc − 50. We also adopt a dropout of 0.7. During the first 100 iterations of training, we set λ = 0 so that no unsupervised-data adaptation is used while still updating Î i,c . Note that Î (ut) i,c are the inferred labels for unsupervised data, and not random at each iteration. Beginning with the 101th iteration, we set λ = {0.1, 1.0} (chosen by cross-validation), and the model typically converges within 2000 to 5000 iterations. We implement ReViSE in TensorFlow [1]. We use Adam [3] for optimization with minibatches of size 1024. We Class 1